Unit for and method of image conversion

ABSTRACT

An image conversion unit ( 200 ) for converting a first image with a first resolution into a second image with a second resolution being higher than the first resolution, comprises a filter ( 112 ) to provide filtered pixel values of the first image to a coefficient-calculating means ( 106 ) for calculating a first filter coefficient on basis of the filtered pixel values. The coefficient-calculating means ( 106 ) is arranged to control an adaptive filtering means ( 104 ) for calculating a second pixel value of the second image on basis of a first pixel value of the first image and the first filter coefficient.

The invention relates to an image conversion unit for converting a firstimage with a first resolution into a second image with a secondresolution, the image conversion unit comprising:

-   -   a coefficient-calculating means for calculating a first filter        coefficient on basis of pixel values of the first image;    -   an adaptive filtering means for calculating a second pixel value        of the second image on basis of a first one of the pixel values        of the first image and the first filter coefficient.

The invention further relates to a method of converting a first imagewith a first resolution into a second image with a second resolution,the method comprising:

-   -   calculating a first filter coefficient on basis of pixel values        of the first image;    -   calculating a second pixel value of the second image on basis of        a first one of the pixel values of the first image and the first        filter coefficient.

The invention further relates to an image processing apparatuscomprising:

-   -   receiving means for receiving a signal corresponding to the        first image; and    -   the above mentioned image conversion unit for converting a first        image into a second image.

The advent of HDTV emphasizes the need for spatial up-conversiontechniques that enable standard definition (SD) video material to beviewed on high definition (HD) television (TV) displays. Conventionaltechniques are linear interpolation methods such as bi-linearinterpolation and methods using poly-phase low-pass interpolationfilters. The former is not popular in television applications because ofits low quality, but the latter is available in commercially availableICs. With the linear methods, the number of pixels in the frame isincreased, but the high frequency part of the spectrum is not extended,i.e. the perceived sharpness of the image is not increased. In otherwords, the capability of the display is not fully exploited.

Additional to the conventional linear techniques, a number of non-linearalgorithms have been proposed to achieve this up-conversion. Sometimesthese techniques are referred to as content-based or edge dependentspatial up-conversion. Some of the techniques are already available onthe consumer electronics market.

An embodiment of the image conversion unit of the kind described in theopening paragraph is known from the article “New Edge-DirectedInterpolation”, by Xin Li et al., in IEEE Transactions on ImageProcessing, Vol. 10, No 10, October 2001, pp. 1521-1527. In this imageconversion unit, the filter coefficients of an interpolationup-conversion filter are adapted to the local image content. Theinterpolation up-conversion filter aperture uses a fourth orderinterpolation algorithm as specified in Equation 1: $\begin{matrix}{{F_{HD}\left( {{2\left( {i + 1} \right)},{2\left( {j + 1} \right)}} \right)} = {\sum\limits_{k = 0}^{1}{\sum\limits_{l = 0}^{1}{w_{{2k} + l}{F_{SD}\left( {{{2i} + {2k}},{{2j} + {2l}}} \right)}}}}} & (1)\end{matrix}$with F_(HD)(i, j) the luminance values of the ID output pixels,F_(SD)(i, j) the luminance values of the input pixels and w_(i) thefilter coefficients. The filter coefficients are obtained from a largeraperture using a Least Mean Squares (LMS) optimization procedure. In thecited article is explained how the filter coefficients are calculated.The method according to the prior art is also explained in connectionwith FIG. 1A and FIG. 1B. The method aims at interpolating along edgesrather than across them to prevent blurring. The authors make thesensible assumption that edge orientation does not change with scaling.Therefore, the coefficients can be approximated from the SD input imagewithin a local window by using the LMS method.

Although the “New Edge-Directed Interpolation” method according to thecited prior art works relatively well in many image parts, there is aproblem with the up-conversion in areas of the image with abundantdetail.

It is an object of the invention to provide an image conversion unit ofthe kind described in the opening paragraph with an improved performancein areas of the image with abundant detail.

This object of the invention is achieved in that the image conversionunit further comprises a further filtering means for filtering the firstimage resulting in filtered pixel values and the coefficient-calculatingmeans being arranged to calculate the first filter coefficient on basisof the filtered pixel values. Pixel values are luminance values or colorvalues. An assumption underlying the prior art algorithm beingimplemented by means of an image conversion unit, i.e. that edgeorientation does not change with scaling, is no longer valid in theseareas. This is because, although the coefficients are optimized on alower density grid, there are no means to prevent aliasing at this grid.In the image conversion unit according to the invention a furtherfiltering means prior to calculating the filter coefficients isincluded. Notice that this further filtering means is not in the directpath of processing the input pixels of the first image into outputpixels, i.e. the pixels of the second image, but in the control path todetermine the filter coefficients. Preferably the further filteringmeans comprises a spatial low-pass filter.

In an embodiment of the image conversion unit according to theinvention, the spatial low-pass filter has a pass-band substantiallycorresponding to a quarter of a sampling frequency of the first image.This low-pass filter is in correspondence with the Sampling Theorem.

The spatial low-pass filter might be a one-dimensional filter or acascade of two orthogonal filters but alternatively a two-dimensionalfilter is applied. In an embodiment of the image conversion unitaccording to the invention, the spatial low-pass filter is arranged tocalculate a first one of the filtered pixel values by means of averagingpixel values of a block of pixels of the first image. An advantage ofthis embodiment according to the invention is that it is relativelysimple. The block of pixels might correspond to e.g. 2*2 or 2*3 or 4*4pixels.

It is a further object of the invention to provide a method of the kinddescribed in the opening paragraph with an improved performance in areasof the image with abundant detail.

This object of the invention is achieved in that the method furthercomprises filtering the first image resulting in filtered pixel valuesand calculating the first filter coefficient on basis of the filteredpixel values.

It is a further object of the invention to provide an image processingapparatus of the kind described in the opening paragraph with animproved performance in areas of the image with abundant detail.

This object of the invention is achieved in that the image conversionunit of the image processing apparatus further comprises a furtherfiltering means for filtering the first image resulting in filteredpixel values and the coefficient-calculating means being arranged tocalculate the first filter coefficient on basis of the filtered pixelvalues. The image processing apparatus optionally comprises a displaydevice for displaying the second image. The image processing apparatusmight e.g. be a TV, a set top box, a VCR (Video Cassette Recorder)player or a DVD (Digital Versatile Disk) player.

Modifications of image conversion unit and variations thereof maycorrespond to modifications and variations thereof of the method and ofthe image processing apparatus described.

These and other aspects of the image conversion unit, of the method andof the image processing apparatus according to the invention will becomeapparent from and will be elucidated with respect to the implementationsand embodiments described hereinafter and with reference to theaccompanying drawings, wherein:

FIG. 1A schematically shows an embodiment of the image conversion unitaccording to the prior art;

FIG. 1B schematically shows a number of pixels to explain the methodaccording to the prior art;

FIG. 2A schematically shows a number of pixels to explain the methodaccording to the invention;

FIG. 2B schematically shows an embodiment of the image conversion unitaccording to the invention;

FIG. 3A schematically shows as SD input image;

FIG. 3B schematically shows the SD input image of FIG. 3A on whichpixels are added in order to increase the resolution;

FIG. 3C schematically shows the image of FIG. 3B after being rotatedover 45 degrees;

FIG. 3D schematically shows an HD output image derived from the SD inputimage of FIG. 3A; and

FIG. 4 schematically shows an embodiment of the image processingapparatus according to the invention.

Same reference numerals are used to denote similar parts throughout thefigures.

FIG. 1A schematically shows an embodiment of the image conversion unit100 according to the prior art. The image conversion unit 100 isprovided with standard definition (SD) images at the input connector 108and provides high definition (HD) images at the output connector 110.The image conversion unit 100 comprises:

A pixel acquisition unit 102 which is arranged to acquire a first set ofvalues of pixels 1-4 in a first neighborhood of a particular locationwithin a first one of the SD input images which corresponds with thelocation of an HD output pixel and to acquire a second set of values ofpixels 1-16 in a second neighborhood of the particular location withinthe first one of the SD input images;

A filter coefficient-calculating unit 106, which is arranged tocalculate filter coefficients on basis the first set of values of pixels1-4 and the second set of values of pixels 1-16. In other words, thefilter coefficients are approximated from the SD input image within alocal window. This is done by using a Least Mean Squares (LMS) methodwhich is explained in connection with FIG. 1B.

An adaptive filtering unit 104 for calculating the value of the HDoutput pixel on basis of the first set of values of pixels 14 and thefilter coefficients as specified with Equation 1. Hence the filtercoefficient-calculating unit 106 is arranged to control the adaptivefiltering unit 104.

FIG. 1B schematically shows a number of pixels 1-16 of an SD input imageand one HD pixel of an HD output image, to explain the method accordingto the prior art. The HD output pixel is interpolated as a weightedaverage of 4 pixels 1-4. That means that the luminance value of the HDoutput pixel F_(HD) results as a weighted sum of its 4 SD neighboringpixels:F _(HD) =w _(i) F _(SD)(1)+w ₂ F _(SD)(2)+w ₃ F _(SD)(3)+w ₄ F_(SD)(4),  (2)where F_(SD)(1) to F_(SD)(4) are the values of the 4 SD input pixels 1-4and w₁ to w₄ are the filter coefficients to be calculated. The authorsof the cited article in which the prior art method is described, makethe sensible assumption that edge orientation does not change withscaling. The consequence of this assumption is that the optimal filtercoefficients are the same as those to interpolate, on the standardresolution grid:

-   -   Pixel 1 from 5, 7, 11, and 4 (that means that pixel 1 can be        derived from its 4 neighbors)    -   Pixel 2 from 6, 8, 3, and 12    -   Pixel 3 from 9, 2, 13, and 15    -   Pixel 4 from 1, 10, 14, and 16        This gives a set of 4 linear equations from which with the        LSM-optimization the optimal 4 filter coefficients to        interpolate the HD output pixel are found.

Denoting M as the pixel set, on the SD-grid, used to calculate the 4weights, the Means Square Error (MSE) over set M in the optimization canbe written as the sum of squared differences between original SD-pixelsF_(SD) and interpolated SD-pixels F_(SI): $\begin{matrix}{{MSE} = {\sum\limits_{F_{{{SD}{({i,j})}} \in M}}\left( {{F_{SD}\left( {{{2i} + 2},{{2j} + 2}} \right)} - {F_{SI}\left( {{{2i} + 2},{{2j} + 2}} \right)}} \right)^{2}}} & (3)\end{matrix}$Which in matrix formulation becomes: $\begin{matrix}{{MSE} = {{\overset{->}{y} - {\overset{->}{w}C}}}^{2}} & (4)\end{matrix}$Here {right arrow over (y)} contains the SD-pixels in M (pixelF_(SD)(1,1) to F_(SD)(1,4), F_(SD)(2,1) to F_(SD)(2,4), F_(SD)(3,1) toF_(SD)(3,4), F_(SD)(4,1) to F_(SD)(4,4) and C is a 4×M² matrix whosek^(th) row is composed of the weighted sum of the four diagonalSD-neighbors of each SD-pixels in {right arrow over (y)}. The weightedsum of each row describes a pixel F_(SI), as used in Equation 3. To findthe minimum MSE, i.e. LMS, the derivation of MSE over {right arrow over(w)} is calculated: $\begin{matrix}{\frac{\partial({MSE})}{\partial\overset{->}{w}} = 0} & (5) \\{{{{- 2}\overset{->}{y}C} + {2\overset{->}{w}C^{2}}} = 0} & (6) \\{\overset{->}{w} = {\left( {C^{T}C} \right)^{- 1}\left( {C^{T}\overset{->}{y}} \right)}} & (7)\end{matrix}$By solving Equation 7 the filter coefficients are found and by usingEquation 2 the values of the HID output pixels can be calculated.

In this example a window of 4 by 4 pixels is used for the calculation ofthe filter coefficients. An LMS optimization on a larger window, e.g. 8by 8 instead of 4 by 4 gives better results.

To further elucidate the invention a very simple example implementationof the invention is explained, in which the further filtering means is a4-pixel averaging filter and the LMS-method is applied on a 5 by 5 blockaround the HD pixel to be interpolated. Reference is made to FIG. 2A forthe numbering of pixels. First, the filtered pixels on the low-densitygrid are calculated, using the average values of their diagonalneighbors, i.e.:

-   -   Pixel a=(5+6+9+1)/4    -   Pixel b=(6+7+1+2)/4    -   . . .    -   . . .    -   Pixel i=(4+12+15+16)/4        Next, the optimal filter coefficients to interpolate pixel HD        (at the same position as pixel e) as a weighted sum of pixels 1,        2, 3, and 4 have to be calculated. To that end, the LMS method        for solving the optimal interpolation of the 9 filtered pixels,        a, b, c, . . . i from their 4 diagonal neighbors is applied. For        example, pixel e is interpolated from pixels a, c, g, and i.        FIG. 2A would have to be extended, as the diagonal neighbors of        pixels a, b, c, d, f, g, h, and I are not shown in this Fig.

Clearly, the aperture for the LMS algorithm can be extended to e.g. 5 by5, or 7 by 7, and a higher order filter can advantageously replace the4-pixel averaging filter. Also a symmetrical filter—with an odd numberof taps—can be used with block sizes of 4 by 4, 8 by 8, etc. Besidesthat the shape of the aperture does not have to be rectangular.

FIG. 2B schematically shows an embodiment of the image conversion unit200 according to the invention. The image conversion unit 200 isprovided with standard definition (SD) images at the input connector 108and provides high definition (ED) images at the output connector 110.The SD input images have pixel matrices as specified in CCIR-601, e.g.625*720 pixels or 525*720 pixels. The HD output images have pixelmatrices with e.g. twice or one-and-a-halve times the number of pixelsin horizontal and vertical direction. The image conversion unit 200comprises:

A pixel acquisition unit 102 which is arranged to acquire a first set ofvalues of pixels 14 in a first neighborhood of a particular locationwithin a first one of the SD input images which corresponds with thelocation of the output pixel HD and to acquire a second set of values ofpixels 1-16 in a second neighborhood of the particular;

A low-pass filter 112 for calculating a set of averaged values a, b, c,. . . i, i.e. filtered pixel values a, b, c, . . . i, on basis of thesecond set of values of pixels 1-16;

A filter coefficient-calculating unit 106 which is arranged to calculatefilter coefficients on basis of the set of filtered pixels a, b, c, . .. i. and the second set of values of pixels 1-16. In other words, thefilter coefficients are approximated from the filtered SD input imagewithin a local window. This is done by using a Least Mean Squares (LMS)method which is explained in connection with FIG. 1B and FIG. 2A; and

An adaptive filtering unit 104 for calculating a pixel value of an HDoutput image on basis of the second set of values of pixels 1-4. The HDoutput pixel is calculated as the weighted sum of the pixels 1-4.

The pixel acquisition unit 102, the low-pass filter 112, the filtercoefficient-calculating unit 106 and the adaptive filtering unit 104 maybe implemented using one processor. Normally, these functions areperformed under control of a software program product. During execution,normally the software program product is loaded into a memory, like aRAM, and executed from there. The program may be loaded from abackground memory, like a ROM, hard disk, or magnetically and/or opticalstorage, or may be loaded via a network like Internet. Optionally anapplication specific integrated circuit provides the disclosedfunctionality.

To convert an SD input image into an HD output image a number ofprocessing steps are needed. By means of FIGS. 3A-3D these processingsteps are explained. FIG. 3A schematically shows an SD input image; FIG.3D schematically shows an HD output image derived from the SD inputimage of FIG. 3A and FIGS. 3B and 3C schematically show intermediateresults.

FIG. 3A schematically shows an SD input image. Each X-sign correspondwith a respective pixel.

FIG. 3B schematically shows the SD input image of FIG. 3A on whichpixels are added in order to increase the resolution. The added pixelsare indicated with +− signs. These added pixels are calculated by meansof interpolation of the respective diagonal neighbors. The filtercoefficients for the interpolation are determined as described inconnection with FIG. 2B.

FIG. 3C schematically shows the image of FIG. 3B after being rotatedover 45 degrees. The same image conversion unit 200 as being applied tocalculate the image as depicted in FIG. 3B on basis of FIG. 3A can beused to calculate the image as shown in FIG. 3D on basis of the image asdepicted in FIG. 3B. That means that new pixel values are calculated bymeans of interpolation of the respective diagonal neighbors.

FIG. 3D schematically shows the final HD output image. The pixels thathave been added in the last conversion step are indicated with o-signs.

FIG. 4 schematically shows an embodiment of the image processingapparatus 400 according to the invention, comprising:

Receiving means 402 for receiving a signal representing SD images. Thesignal may be a broadcast signal received via an antenna or cable butmay also be a signal from a storage device like a VCR (Video CassetteRecorder) or Digital Versatile Disk (DVD). The signal is provided at theinput connector 408;

The image conversion unit 404 as described in connection with FIG. 2B;and

A display device 406 for displaying the HD output images of the imageconversion unit 200. This display device 406 is optional.

The image processing apparatus 400 might e.g. be a TV. Alternatively theimage processing apparatus 400 does not comprise the optional displaydevice but provides HD images to an apparatus that does comprise adisplay device 406. Then the image processing apparatus 400 might bee.g. a set top box, a satellite-tuner, a VCR player or a DVD player. Butit might also be a system being applied by a film-studio or broadcaster.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention and that those skilled in the art willbe able to design alternative embodiments without departing from thescope of the appended claims. In the claims, any reference signs placedbetween parentheses shall not be constructed as limiting the claim. Theword ‘comprising’ does not exclude the presence of elements or steps notlisted in a claim. The word “a” or “an” preceding an element does notexclude the presence of a plurality of such elements. The invention canbe implemented by means of hardware comprising several distinct elementsand by means of a suitable programmed computer. In the unit claimsenumerating several means, several of these means can be embodied by oneand the same item of hardware.

1. An image conversion unit for converting a first image with a firstresolution into a second image with a second resolution, the imageconversion unit comprising: a coefficient-calculating means forcalculating a first filter coefficient on basis of pixel values of thefirst image; an adaptive filtering means for calculating a second pixelvalue of the second image on basis of a first one of the pixel values ofthe first image and the first filter coefficient, characterized in thatthe image conversion unit further comprises a further filtering meansfor filtering the first image resulting in filtered pixel values and thecoefficient-calculating means being arranged to calculate the firstfilter coefficient on basis of the filtered pixel values.
 2. An imageconversion unit as claimed in claim 1, characterized in that the furtherfiltering means comprises a spatial low-pass filter.
 3. An imageconversion unit as claimed in claim 2, characterized in that the spatiallow-pass filter has a pass-band substantially corresponding to a quarterof a sampling frequency of the first image.
 4. An image conversion unitas claimed in claim 2, characterized in that the spatial low-pass filteris arranged to calculate a first one of the filtered pixel values bymeans of averaging pixel values of a block of pixels of the first image.5. A method of converting a first image with a first resolution into asecond image with a second resolution, the method comprising:calculating a first filter coefficient on basis of pixel values of thefirst image; calculating a second pixel value of the second image onbasis of a first one of the pixel values of the first image and thefirst filter coefficient, characterized in that the method furthercomprises filtering the first image resulting in filtered pixel valuesand calculating the first filter coefficient on basis of the filteredpixel values.
 6. An image processing apparatus comprising: receivingmeans for receiving a signal corresponding to a first image; and theimage conversion unit for converting the first image into a secondimage, as claimed in claim
 1. 7. An image processing apparatus asclaimed in claim 6, further comprising a display device for displayingthe second image.